Expected Eligibility Traces

The question of how to determine which states and actions are responsible for a certain outcome is known as the credit assignment problem remains central research in reinforcement learning artificial intelligence. Eligibility traces enable efficient recent sequence experienced by agent, but not counterfactual sequences that could also have led current state. In this work, we introduce expected eligibility traces. Expected allow, with single update, update preceded state, even if they did do so on occasion. We discuss when provide benefits over classic (instantaneous) temporal-difference learning, show some- times substantial improvements can be attained. way smoothly interpolate between instantaneous mechanism similar bootstrapping, ensures resulting algorithm strict generalisation TD(?). Finally, possible extensions connections related ideas, such successor features.